Integration of Image Capture and Processing { Beyond Single Chip Digital Camera

An important trend in the design of digital cameras is the integration of capture and processing onto a single CMOS chip. Although integrating the components of a digital camera system onto a single chip signi cantly reduces system size and power, it does not fully exploit the potential advantages of integration. We argue that a key advantage of integration is the ability to exploit the high speed imaging capability of CMOS image sensors to enable new applications such as multiple capture for enhancing dynamic range and to improve the performance of existing applications such as optical ow estimation. Conventional digital cameras operate at low frame rates and it would be too costly, if not infeasible, to operate their chips at high frame rates. Integration solves this problem. The idea is to capture images at much higher frame rates than the standard frame rate, process the high frame rate data on chip, and output the video sequence and the application speci c data at standard frame rate. This idea is applied to optical ow estimation, where signi cant performance improvements are demonstrated over methods using standard frame rate sequences. We then investigate the constraints on memory size and processing power that can be integrated with a CMOS image sensor in a 0:18 m process and below. We show that enough memory and processing power can be integrated to be able to not only perform the functions of a conventional camera system but also to perform applications such as real time optical ow estimation.